{ "cells": [ { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "# Entity Service Similarity Scores Output\n", "\n", "This tutorial demonstrates generating CLKs from PII, creating a new project on the entity service, and how to retrieve the results. \n", "The output type is raw similarity scores. This output type is particularly useful for determining a good threshold for the greedy solver used in mapping.\n", "\n", "The sections are usually run by different participants - but for illustration all is carried out in this one file. The participants providing data are *Alice* and *Bob*, and the analyst is acting as the integration authority.\n", "\n", "### Who learns what?\n", "\n", "Alice and Bob will both generate and upload their CLKs.\n", "\n", "The analyst - who creates the linkage project - learns the `similarity scores`. Be aware that this is a lot of information and are subject to frequency attacks.\n", "\n", "### Steps\n", "\n", "* Check connection to Entity Service\n", "* Data preparation\n", " * Write CSV files with PII\n", " * Create a Linkage Schema\n", "* Create Linkage Project\n", "* Generate CLKs from PII\n", "* Upload the PII\n", "* Create a run\n", "* Retrieve and analyse results" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [], "source": [ "%matplotlib inline\n", "\n", "import json\n", "import os\n", "import time\n", "import pandas as pd\n", "\n", "import matplotlib.pyplot as plt\n", "import requests\n", "import anonlinkclient.rest_client\n", "from IPython.display import clear_output" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Check Connection\n", "\n", "If you are connecting to a custom entity service, change the address here." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Testing anonlink-entity-service hosted at https://anonlink.easd.data61.xyz\n" ] } ], "source": [ "url = os.getenv(\"SERVER\", \"https://anonlink.easd.data61.xyz\")\n", "print(f'Testing anonlink-entity-service hosted at {url}')" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\"project_count\": 845, \"rate\": 593838, \"status\": \"ok\"}\r\n" ] } ], "source": [ "!anonlink status --server \"{url}\"" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Data preparation\n", "\n", "Following the [anonlink client command line tutorial](https://anonlink-client.readthedocs.io/en/latest/tutorial/tutorial_cli.html) we will use a dataset from the `recordlinkage` library. We will just write both datasets out to temporary CSV files.\n", "\n", "If you are following along yourself you may have to adjust the file names in all the `!anonlink` commands." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [], "source": [ "from tempfile import NamedTemporaryFile\n", "from recordlinkage.datasets import load_febrl4" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
given_namesurnamestreet_numberaddress_1address_2suburbpostcodestatedate_of_birthsoc_sec_id
rec_id
rec-1070-orgmichaelaneumann8stanley streetmiamiwinston hills4223nsw191511115304218
rec-1016-orgcourtneypainter12pinkerton circuitbega flatsrichlands4560vic191612144066625
rec-4405-orgcharlesgreen38salkauskas crescentkeladapto4566nsw194809304365168
\n", "
" ], "text/plain": [ " given_name surname street_number address_1 \\\n", "rec_id \n", "rec-1070-org michaela neumann 8 stanley street \n", "rec-1016-org courtney painter 12 pinkerton circuit \n", "rec-4405-org charles green 38 salkauskas crescent \n", "\n", " address_2 suburb postcode state date_of_birth \\\n", "rec_id \n", "rec-1070-org miami winston hills 4223 nsw 19151111 \n", "rec-1016-org bega flats richlands 4560 vic 19161214 \n", "rec-4405-org kela dapto 4566 nsw 19480930 \n", "\n", " soc_sec_id \n", "rec_id \n", "rec-1070-org 5304218 \n", "rec-1016-org 4066625 \n", "rec-4405-org 4365168 " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dfA, dfB = load_febrl4()\n", "\n", "a_csv = NamedTemporaryFile('w')\n", "a_clks = NamedTemporaryFile('w', suffix='.json')\n", "dfA.to_csv(a_csv)\n", "a_csv.seek(0)\n", "\n", "b_csv = NamedTemporaryFile('w')\n", "b_clks = NamedTemporaryFile('w', suffix='.json')\n", "dfB.to_csv(b_csv)\n", "b_csv.seek(0)\n", "\n", "dfA.head(3)" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Schema Preparation\n", "\n", "The linkage schema must be agreed on by the two parties. A hashing schema instructs clkhash how to treat each column for generating CLKs. A detailed description of the hashing schema can be found in the api docs. We will ignore the columns `rec_id` and `soc_sec_id` for CLK generation." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [], "source": [ "schema = NamedTemporaryFile('wt')" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting /tmp/tmprrzuvk7f\n" ] } ], "source": [ "%%writefile {schema.name}\n", "{\n", " \"version\": 3,\n", " \"clkConfig\": {\n", " \"l\": 1024,\n", " \"xor_folds\": 0,\n", " \"kdf\": {\n", " \"type\": \"HKDF\",\n", " \"hash\": \"SHA256\",\n", " \"info\": \"c2NoZW1hX2V4YW1wbGU=\",\n", " \"salt\": \"SCbL2zHNnmsckfzchsNkZY9XoHk96P/G5nUBrM7ybymlEFsMV6PAeDZCNp3rfNUPCtLDMOGQHG4pCQpfhiHCyA==\",\n", " \"keySize\": 64\n", " }\n", " },\n", " \"features\": [\n", " {\n", " \"identifier\": \"rec_id\",\n", " \"ignored\": true\n", " },\n", " {\n", " \"identifier\": \"given_name\",\n", " \"format\": {\n", " \"type\": \"string\",\n", " \"encoding\": \"utf-8\"\n", " },\n", " \"hashing\": {\n", " \"strategy\": {\n", " \"bitsPerFeature\": 200\n", " },\n", " \"hash\": {\n", " \"type\": \"doubleHash\"\n", " },\n", " \"comparison\": {\n", " \"type\": \"ngram\",\n", " \"n\": 2,\n", " \"positional\": false\n", " }\n", " }\n", " },\n", " {\n", " \"identifier\": \"surname\",\n", " \"format\": {\n", " \"type\": \"string\",\n", " \"encoding\": \"utf-8\"\n", " },\n", " \"hashing\": {\n", " \"strategy\": {\n", " \"bitsPerFeature\": 200\n", " },\n", " \"hash\": {\n", " \"type\": \"doubleHash\"\n", " },\n", " \"comparison\": {\n", " \"type\": \"ngram\",\n", " \"n\": 2,\n", " \"positional\": false\n", " }\n", " }\n", " },\n", " {\n", " \"identifier\": \"street_number\",\n", " \"format\": {\n", " \"type\": \"integer\"\n", " },\n", " \"hashing\": {\n", " \"missingValue\": {\n", " \"sentinel\": \"\"\n", " },\n", " \"strategy\": {\n", " \"bitsPerFeature\": 100\n", " },\n", " \"hash\": {\n", " \"type\": \"doubleHash\"\n", " },\n", " \"comparison\": {\n", " \"type\": \"ngram\",\n", " \"n\": 1,\n", " \"positional\": true\n", " }\n", " }\n", " },\n", " {\n", " \"identifier\": \"address_1\",\n", " \"format\": {\n", " \"type\": \"string\",\n", " \"encoding\": \"utf-8\"\n", " },\n", " \"hashing\": {\n", " \"strategy\": {\n", " \"bitsPerFeature\": 100\n", " },\n", " \"hash\": {\n", " \"type\": \"doubleHash\"\n", " },\n", " \"comparison\": {\n", " \"type\": \"ngram\",\n", " \"n\": 2,\n", " \"positional\": false\n", " }\n", " }\n", " },\n", " {\n", " \"identifier\": \"address_2\",\n", " \"format\": {\n", " \"type\": \"string\",\n", " \"encoding\": \"utf-8\"\n", " },\n", " \"hashing\": {\n", " \"strategy\": {\n", " \"bitsPerFeature\": 100\n", " },\n", " \"hash\": {\n", " \"type\": \"doubleHash\"\n", " },\n", " \"comparison\": {\n", " \"type\": \"ngram\",\n", " \"n\": 2,\n", " \"positional\": false\n", " }\n", " }\n", " },\n", " {\n", " \"identifier\": \"suburb\",\n", " \"format\": {\n", " \"type\": \"string\",\n", " \"encoding\": \"utf-8\"\n", " },\n", " \"hashing\": {\n", " \"strategy\": {\n", " \"bitsPerFeature\": 100\n", " },\n", " \"hash\": {\n", " \"type\": \"doubleHash\"\n", " },\n", " \"comparison\": {\n", " \"type\": \"ngram\",\n", " \"n\": 2,\n", " \"positional\": false\n", " }\n", " }\n", " },\n", " {\n", " \"identifier\": \"postcode\",\n", " \"format\": {\n", " \"type\": \"integer\",\n", " \"minimum\": 100,\n", " \"maximum\": 9999\n", " },\n", " \"hashing\": {\n", " \"strategy\": {\n", " \"bitsPerFeature\": 100\n", " },\n", " \"hash\": {\n", " \"type\": \"doubleHash\"\n", " },\n", " \"comparison\": {\n", " \"type\": \"ngram\",\n", " \"n\": 1,\n", " \"positional\": true\n", " }\n", " }\n", " },\n", " {\n", " \"identifier\": \"state\",\n", " \"format\": {\n", " \"type\": \"string\",\n", " \"encoding\": \"utf-8\",\n", " \"maxLength\": 3\n", " },\n", " \"hashing\": {\n", " \"strategy\": {\n", " \"bitsPerFeature\": 100\n", " },\n", " \"hash\": {\n", " \"type\": \"doubleHash\"\n", " },\n", " \"comparison\": {\n", " \"type\": \"ngram\",\n", " \"n\": 2,\n", " \"positional\": false\n", " }\n", " }\n", " },\n", " {\n", " \"identifier\": \"date_of_birth\",\n", " \"format\": {\n", " \"type\": \"integer\"\n", " },\n", " \"hashing\": {\n", " \"missingValue\": {\n", " \"sentinel\": \"\"\n", " },\n", " \"strategy\": {\n", " \"bitsPerFeature\": 200\n", " },\n", " \"hash\": {\n", " \"type\": \"doubleHash\"\n", " },\n", " \"comparison\": {\n", " \"type\": \"ngram\",\n", " \"n\": 1,\n", " \"positional\": true\n", " }\n", " }\n", " },\n", " {\n", " \"identifier\": \"soc_sec_id\",\n", " \"ignored\": true\n", " }\n", " ]\n", "}" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Create Linkage Project\n", "\n", "The analyst carrying out the linkage starts by creating a linkage project of the desired output type with the Entity Service.\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Credentials will be saved in /tmp/tmp1h8qppks\n", "\u001b[31mProject created\u001b[0m\n" ] }, { "data": { "text/plain": [ "{'project_id': '8f8347b3e97665ebc87f4a1744a2a62e0ae4c999184bc754',\n", " 'result_token': 'f6ef6b121f3e5861bceddc36cf1cfdebf9c25a2352937c90',\n", " 'update_tokens': ['192e38e1f9e773b945c882799e5490502b9454c711b66e2d',\n", " 'f1ca0cbbdc3055c731e898a4ebe6121be9e8d82541fb78fd']}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "creds = NamedTemporaryFile('wt')\n", "print(\"Credentials will be saved in\", creds.name)\n", "\n", "!anonlink create-project \\\n", " --schema \"{schema.name}\" \\\n", " --output \"{creds.name}\" \\\n", " --type \"similarity_scores\" \\\n", " --server \"{url}\"\n", "\n", "creds.seek(0)\n", "\n", "with open(creds.name, 'r') as f:\n", " credentials = json.load(f)\n", "\n", "project_id = credentials['project_id']\n", "credentials" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "**Note:** the analyst will need to pass on the `project_id` (the id of the linkage project) and one of the two `update_tokens` to each data provider.\n", "\n", "## Hash and Upload\n", "\n", "At the moment both data providers have *raw* personally identiy information. We first have to generate CLKs from the raw entity information. Please see [clkhash](https://clkhash.readthedocs.io/) documentation for further details on this." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[31mCLK data written to /tmp/tmp63vp_3mj.json\u001b[0m\n", "\u001b[31mCLK data written to /tmp/tmpr4cqqglj.json\u001b[0m\n" ] } ], "source": [ "!anonlink hash \"{a_csv.name}\" secret \"{schema.name}\" \"{a_clks.name}\"\n", "!anonlink hash \"{b_csv.name}\" secret \"{schema.name}\" \"{b_clks.name}\"" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "Now the two clients can upload their data providing the appropriate *upload tokens*." ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "### Alice uploads her data" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [], "source": [ "with NamedTemporaryFile('wt') as f:\n", " !anonlink upload \\\n", " --project=\"{project_id}\" \\\n", " --apikey=\"{credentials['update_tokens'][0]}\" \\\n", " --server \"{url}\" \\\n", " --output \"{f.name}\" \\\n", " \"{a_clks.name}\"\n", " res = json.load(open(f.name))\n", " alice_receipt_token = res['receipt_token']" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "Every upload gets a receipt token. In some operating modes this receipt is required to access the results." ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "### Bob uploads his data" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [], "source": [ "with NamedTemporaryFile('wt') as f:\n", " !anonlink upload \\\n", " --project=\"{project_id}\" \\\n", " --apikey=\"{credentials['update_tokens'][1]}\" \\\n", " --server \"{url}\" \\\n", " --output \"{f.name}\" \\\n", " \"{b_clks.name}\"\n", " \n", " bob_receipt_token = json.load(open(f.name))['receipt_token']" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Create a run\n", "\n", "Now the project has been created and the CLK data has been uploaded we can carry out some privacy preserving record linkage. Try with a few different threshold values:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [], "source": [ "with NamedTemporaryFile('wt') as f:\n", " !anonlink create \\\n", " --project=\"{project_id}\" \\\n", " --apikey=\"{credentials['result_token']}\" \\\n", " --server \"{url}\" \\\n", " --threshold 0.75 \\\n", " --output \"{f.name}\"\n", " \n", " run_id = json.load(open(f.name))['run_id']" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Results\n", "\n", "Now after some delay (depending on the size) we can fetch the result.\n", "This can be done with `anonlink`:\n", "\n", " !anonlink results --server \"{url}\" \\\n", " --project=\"{credentials['project_id']}\" \\\n", " --apikey=\"{credentials['result_token']}\" --output results.txt\n", " \n", "However for this tutorial we are going to use the `anonlinkclient.rest_client` module:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "State: completed\n", "Stage (2/2): compute similarity scores\n", "Progress: 100.00%\n" ] } ], "source": [ "from anonlinkclient.rest_client import RestClient\n", "from anonlinkclient.rest_client import format_run_status\n", "rest_client = RestClient(url)\n", "for update in rest_client.watch_run_status(project_id, run_id, credentials['result_token'], timeout=300):\n", " clear_output(wait=True)\n", " print(format_run_status(update))" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "data = json.loads(rest_client.run_get_result_text(\n", " project_id, \n", " run_id, \n", " credentials['result_token']))['similarity_scores']" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "This result is a large list of tuples recording the similarity between all rows above the given threshold." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[0, 76], [1, 2345], 1.0]\n", "[[0, 83], [1, 3439], 1.0]\n", "[[0, 103], [1, 863], 1.0]\n", "[[0, 154], [1, 2391], 1.0]\n", "[[0, 177], [1, 4247], 1.0]\n", "[[0, 192], [1, 1176], 1.0]\n", "[[0, 270], [1, 4516], 1.0]\n", "[[0, 312], [1, 1253], 1.0]\n", "[[0, 407], [1, 3743], 1.0]\n", "[[0, 670], [1, 3550], 1.0]\n" ] } ], "source": [ "for row in data[:10]:\n", " print(row)" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "Note there can be a lot of similarity scores:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "data": { "text/plain": [ "280116" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(data)" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "We will display a *sample* of these similarity scores in a histogram using matplotlib:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEGCAYAAACQO2mwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAXVklEQVR4nO3dfbRddX3n8ffHpKC15UmyWDahJqOZ2minFbOQqWPLkgoBi6EdcKAdjTQ1taLT6TC1MM4Sl0qXLrVYrNqyJCUw1oiMHaJFmZQHnToDEkQeggVug0oiSiSAtT5g9Dt/nF/qMdxfcnPPzb1J7vu11ll37+/+7b1/v9yHT/bD2SdVhSRJ43nSTHdAkrTvMiQkSV2GhCSpy5CQJHUZEpKkrrkz3YGpduSRR9bChQtnuhuStF+59dZbv1FV83auH3AhsXDhQjZs2DDT3ZCk/UqSL49X93STJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSp64B7x/UoTj336nHrH3/38mnuiSTtGzySkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV27DYkkq5M8lOSuodo7k/xDkjuS/E2Sw4aWnZ9kLMk9SU4aqi9rtbEk5w3VFyW5udU/kuSgVj+4zY+15QunatCSpImZyJHEZcCynWrrgedW1b8B7gXOB0iyBDgTeE5b5/1J5iSZA7wPOBlYApzV2gK8A7ioqp4FPAKsbPWVwCOtflFrJ0maRrsNiar6DLBtp9r/rqrtbfYmYEGbXg6srarvVdX9wBhwbHuNVdWmqnocWAssTxLgxcBVbf01wGlD21rTpq8CTmjtJUnTZCquSfwO8Mk2PR94YGjZ5lbr1Z8GPDoUODvqP7attvyx1v4JkqxKsiHJhq1bt448IEnSwEghkeSNwHbgQ1PTncmpqkuqamlVLZ03b95MdkWSDihzJ7tiklcBvw6cUFXVyluAo4eaLWg1OvWHgcOSzG1HC8Ptd2xrc5K5wKGtvSRpmkzqSCLJMuANwMuq6ttDi9YBZ7Y7kxYBi4HPAbcAi9udTAcxuLi9roXLDcDpbf0VwNVD21rRpk8Hrh8KI0nSNNjtkUSSDwPHA0cm2QxcwOBupoOB9e1a8k1V9Zqq2pjkSuBuBqehzqmqH7TtvA64FpgDrK6qjW0XfwysTfI24Dbg0la/FLgiyRiDC+dnTsF4JUl7YLchUVVnjVO+dJzajvYXAheOU78GuGac+iYGdz/tXP8ucMbu+idJ2nt8x7UkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVLXbkMiyeokDyW5a6h2RJL1Se5rXw9v9SS5OMlYkjuSHDO0zorW/r4kK4bqz09yZ1vn4iTZ1T4kSdNnIkcSlwHLdqqdB1xXVYuB69o8wMnA4vZaBXwABn/wgQuAFwDHAhcM/dH/APDqofWW7WYfkqRpstuQqKrPANt2Ki8H1rTpNcBpQ/XLa+Am4LAkTwdOAtZX1baqegRYDyxryw6pqpuqqoDLd9rWePuQJE2TyV6TOKqqHmzTXwOOatPzgQeG2m1utV3VN49T39U+JEnTZOQL1+0IoKagL5PeR5JVSTYk2bB169a92RVJmlUmGxJfb6eKaF8favUtwNFD7Ra02q7qC8ap72ofT1BVl1TV0qpaOm/evEkOSZK0s8mGxDpgxx1KK4Crh+qvbHc5HQc81k4ZXQucmOTwdsH6RODatuybSY5rdzW9cqdtjbcPSdI0mbu7Bkk+DBwPHJlkM4O7lN4OXJlkJfBl4OWt+TXAKcAY8G3gbICq2pbkrcAtrd1bqmrHxfDXMriD6inAJ9uLXexDkjRNdhsSVXVWZ9EJ47Qt4JzOdlYDq8epbwCeO0794fH2IUmaPr7jWpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1jRQSSf4wycYkdyX5cJInJ1mU5OYkY0k+kuSg1vbgNj/Wli8c2s75rX5PkpOG6stabSzJeaP0VZK05yYdEknmA/8JWFpVzwXmAGcC7wAuqqpnAY8AK9sqK4FHWv2i1o4kS9p6zwGWAe9PMifJHOB9wMnAEuCs1laSNE1GPd00F3hKkrnATwIPAi8GrmrL1wCntenlbZ62/IQkafW1VfW9qrofGAOOba+xqtpUVY8Da1tbSdI0mXRIVNUW4F3AVxiEw2PArcCjVbW9NdsMzG/T84EH2rrbW/unDdd3WqdXf4Ikq5JsSLJh69atkx2SJGkno5xuOpzB/+wXAT8DPJXB6aJpV1WXVNXSqlo6b968meiCJB2QRjnd9GvA/VW1taq+D3wMeCFwWDv9BLAA2NKmtwBHA7TlhwIPD9d3WqdXlyRNk1FC4ivAcUl+sl1bOAG4G7gBOL21WQFc3abXtXna8uurqlr9zHb30yJgMfA54BZgcbtb6iAGF7fXjdBfSdIemrv7JuOrqpuTXAV8HtgO3AZcAvwtsDbJ21rt0rbKpcAVScaAbQz+6FNVG5NcySBgtgPnVNUPAJK8DriWwZ1Tq6tq42T7K0nac5MOCYCqugC4YKfyJgZ3Ju3c9rvAGZ3tXAhcOE79GuCaUfooSZo833EtSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpa6RnN80Wp5579bj1j7/bD8qTdGDzSEKS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSukYKiSSHJbkqyT8k+WKSf5vkiCTrk9zXvh7e2ibJxUnGktyR5Jih7axo7e9LsmKo/vwkd7Z1Lk6SUforSdozox5J/Bnwqap6NvCLwBeB84DrqmoxcF2bBzgZWNxeq4APACQ5ArgAeAFwLHDBjmBpbV49tN6yEfsrSdoDkw6JJIcCvwJcClBVj1fVo8ByYE1rtgY4rU0vBy6vgZuAw5I8HTgJWF9V26rqEWA9sKwtO6SqbqqqAi4f2pYkaRqMciSxCNgK/FWS25J8MMlTgaOq6sHW5mvAUW16PvDA0PqbW21X9c3j1J8gyaokG5Js2Lp16whDkiQNGyUk5gLHAB+oqucB/8yPTi0B0I4AaoR9TEhVXVJVS6tq6bx58/b27iRp1hglJDYDm6vq5jZ/FYPQ+Ho7VUT7+lBbvgU4emj9Ba22q/qCceqSpGky6ZCoqq8BDyT5uVY6AbgbWAfsuENpBbDjE3vWAa9sdzkdBzzWTktdC5yY5PB2wfpE4Nq27JtJjmt3Nb1yaFuSpGkw6ifTvR74UJKDgE3A2QyC58okK4EvAy9vba8BTgHGgG+3tlTVtiRvBW5p7d5SVdva9GuBy4CnAJ9sL0nSNBkpJKrqC8DScRadME7bAs7pbGc1sHqc+gbguaP0UZI0eb7jWpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqWvkkEgyJ8ltST7R5hcluTnJWJKPJDmo1Q9u82Nt+cKhbZzf6vckOWmovqzVxpKcN2pfJUl7ZiqOJP4A+OLQ/DuAi6rqWcAjwMpWXwk80uoXtXYkWQKcCTwHWAa8vwXPHOB9wMnAEuCs1laSNE1GCokkC4CXAh9s8wFeDFzVmqwBTmvTy9s8bfkJrf1yYG1Vfa+q7gfGgGPba6yqNlXV48Da1laSNE1GPZJ4D/AG4Idt/mnAo1W1vc1vBua36fnAAwBt+WOt/b/Ud1qnV3+CJKuSbEiyYevWrSMOSZK0w6RDIsmvAw9V1a1T2J9JqapLqmppVS2dN2/eTHdHkg4Yc0dY94XAy5KcAjwZOAT4M+CwJHPb0cICYEtrvwU4GticZC5wKPDwUH2H4XV6dUnSNJj0kURVnV9VC6pqIYMLz9dX1W8DNwCnt2YrgKvb9Lo2T1t+fVVVq5/Z7n5aBCwGPgfcAixud0sd1PaxbrL9lSTtuVGOJHr+GFib5G3AbcClrX4pcEWSMWAbgz/6VNXGJFcCdwPbgXOq6gcASV4HXAvMAVZX1ca90F9JUseUhERV3Qjc2KY3Mbgzaec23wXO6Kx/IXDhOPVrgGumoo+SpD3nO64lSV1743TTrHHquVePW//4u307h6QDg0cSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlS16RDIsnRSW5IcneSjUn+oNWPSLI+yX3t6+GtniQXJxlLckeSY4a2taK1vy/JiqH685Pc2da5OElGGawkac+MciSxHTi3qpYAxwHnJFkCnAdcV1WLgevaPMDJwOL2WgV8AAahAlwAvAA4FrhgR7C0Nq8eWm/ZCP2VJO2hSYdEVT1YVZ9v0/8EfBGYDywH1rRma4DT2vRy4PIauAk4LMnTgZOA9VW1raoeAdYDy9qyQ6rqpqoq4PKhbUmSpsGUXJNIshB4HnAzcFRVPdgWfQ04qk3PBx4YWm1zq+2qvnmcuiRpmowcEkl+CvifwH+uqm8OL2tHADXqPibQh1VJNiTZsHXr1r29O0maNUYKiSQ/wSAgPlRVH2vlr7dTRbSvD7X6FuDoodUXtNqu6gvGqT9BVV1SVUuraum8efNGGZIkacjcya7Y7jS6FPhiVf3p0KJ1wArg7e3r1UP11yVZy+Ai9WNV9WCSa4E/GbpYfSJwflVtS/LNJMcxOI31SuC9k+3vdDr13Ku7yz7+7uXT2BNJGs2kQwJ4IfAK4M4kX2i1/8YgHK5MshL4MvDytuwa4BRgDPg2cDZAC4O3Are0dm+pqm1t+rXAZcBTgE+2lyRpmkw6JKrq74He+xZOGKd9Aed0trUaWD1OfQPw3Mn2UZI0Gt9xLUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqWuUB/xpEnpPiPXpsJL2RR5JSJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHV5C+w+wltjJe2LPJKQJHUZEpKkLkNCktTlNYl9nNcqJM2kff5IIsmyJPckGUty3kz3R5Jmk336SCLJHOB9wEuAzcAtSdZV1d0z27OZ5xGGpOmwT4cEcCwwVlWbAJKsBZYDsz4kenrh0WOo6EC1p78LB4K98fu8r4fEfOCBofnNwAt2bpRkFbCqzX4ryT2T3N+RwDcmue5+KX86+8bMLPw+45hnhRF/n58xXnFfD4kJqapLgEtG3U6SDVW1dAq6tN9wzLODY54d9saY9/UL11uAo4fmF7SaJGka7OshcQuwOMmiJAcBZwLrZrhPkjRr7NOnm6pqe5LXAdcCc4DVVbVxL+5y5FNW+yHHPDs45tlhysecqprqbUqSDhD7+ukmSdIMMiQkSV2zJiR293iPJBcl+UJ73Zvk0aFlPxhatt9cOJ/AmH82yQ1JbktyR5JThpad39a7J8lJ09vzyZnseJMsTPKdoe/xX0x/7ydnAmN+RpLr2nhvTLJgaNmKJPe114rp7fnkjTjm/fV3eXWSh5Lc1VmeJBe3f5M7khwztGy073NVHfAvBhe9/xH4V8BBwO3Akl20fz2Di+Q75r8102PYG2NmcJHr99v0EuBLQ9O3AwcDi9p25sz0mPbieBcCd830GPbSmD8KrGjTLwauaNNHAJva18Pb9OEzPaa9OeY2v9/9Lrd+/wpwTO/nFDgF+CQQ4Djg5qn6Ps+WI4l/ebxHVT0O7Hi8R89ZwIenpWd7z0TGXMAhbfpQ4Kttejmwtqq+V1X3A2Nte/uyUca7v5rImJcA17fpG4aWnwSsr6ptVfUIsB5YNg19HtUoY95vVdVngG27aLIcuLwGbgIOS/J0puD7PFtCYrzHe8wfr2GSZzD43/P1Q+UnJ9mQ5KYkp+29bk6piYz5zcB/TLIZuIbBEdRE193XjDJegEXtNNSnk7xor/Z06kxkzLcDv9mmfwP46SRPm+C6+6JRxgz75+/yRPT+XUb+Ps+WkNgTZwJXVdUPhmrPqMFb3X8LeE+SZ85M16bcWcBlVbWAweHqFUkO5J+J3ngfBH62qp4H/Bfgr5Mcsovt7E/+K/CrSW4DfpXBEwt+sOtV9nu7GvOB+ru81xzIfxCG7cnjPc5kp1NNVbWlfd0E3Ag8b+q7OOUmMuaVwJUAVfX/gCczeCja/vg4lEmPt51We7jVb2Vwzvtf7/Uej263Y66qr1bVb7YAfGOrPTqRdfdRo4x5f/1dnojev8vo3+eZviAzTRd95jK4YLOIH13ses447Z4NfIn2JsNWOxw4uE0fCdzHLi567yuviYyZwYWuV7Xpn2dwjj7Ac/jxC9eb2PcvXI8y3nk7xsfggugW4IiZHtMUjflI4Elt+kLgLW36COD+9vN9eJs+0Me8X/4uD41rIf0L1y/lxy9cf26qvs8zPvBp/Ac+BbiXwf8S39hqbwFeNtTmzcDbd1rvl4E72w/jncDKmR7LVI2ZwQW+z7axfQE4cWjdN7b17gFOnumx7M3xAv8e2NhqnwdOnemxTOGYT29/DO8FPrjjj2Rb9jsMbkoYA86e6bHs7THv57/LH2ZwWvT7DK4rrAReA7ymLQ+DD2j7xza2pVP1ffaxHJKkrtlyTUKSNAmGhCSpy5CQJHUZEpKkLkNCktRlSGjWSPLBJEv2oP3SJBe36Vcl+fM93N/w+scn+eU967E08/bpjy+VplJV/e4ett8AbJjMvpLM3Wn944FvAf93MtubCknm1I8/bkbaLY8kdMBJ8tQkf5vk9iR3JfkPrX5jkqVt+ltJ3plkY5K/S3JsW74pyctam+OTfGKc7Z+a5Ob2QMC/S3JUq785yRVJPsvguVDHJ/lEkoUM3vj0h+1zDF6U5P4kP9HWO2R4fmg/Z7T+357kM602J8m7Wv2OJK9v9RNaf+5snz1wcKt/Kck7knweOCPJM5N8KsmtSf5Pkmfvje+BDhweSehAtAz4alW9FCDJoeO0eSpwfVX9UZK/Ad4GvITBu7LXALv6QJq/B46rqkryu8AbgHPbsiXAv6uq7yQ5HqCqvpTBBxl9q6re1fp0I4NHKfwvBs8L+1hVfX+n/bwJOKmqtiQ5rNVWMXg8wy9V1fYkRyR5MnAZcEJV3ZvkcuD3gfe0dR6uqmPafq9j8C7d+5K8AHg/g89ckMblkYQORHcCL2n/g35RVT02TpvHgU8Ntf90+yN9J4M/wruyALg2yZ3AHzF41tUO66rqOxPo4weBs9v02cBfjdPms8BlSV7N4MN2AH4N+Muq2g5QVduAnwPur6p7W5s1DD6kZoePACT5KQaPpvhoki8Afwk8fQJ91SxmSOiA0/5YHsPgD/7bkrxpnGbfrx89k+aHwPfauj9k90fY7wX+vKp+Afg9Bk+T3eGfJ9jHzwIL29HGnKp6wsdSVtVrgP/O4Cmetw59JsKe2tGnJwGPVtUvDb1+fpLb1CxhSOiAk+RngG9X1f8A3skgMKbSofzoccsT/czgfwJ+eqfa5cBfM/5RBEmeWVU3V9WbgK0MwmI98HtJ5rY2RzB4COPCJM9qq74C+PTO26uqbwL3JzmjrZskvzjB/muWMiR0IPoF4HPtlMoFDK43TKU3MzhlcyvwjQmu83HgN3ZcuG61DzF4fHPvo3Lf2S5E38XgrqjbGZym+gpwR5Lbgd+qqu8yOGX10XYK7IfAX3S2+dvAyrbuRg6Aj/bU3uVTYKUZkuR0YHlVvWKm+yL1eHeTNAOSvBc4mcFnI0j7LI8kJEldXpOQJHUZEpKkLkNCktRlSEiSugwJSVLX/wfHVDfj9t4R7gAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.style.use('seaborn-deep')\n", "plt.hist([score for _, _, score in data], bins=50)\n", "plt.xlabel('similarity score')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "The vast majority of these similarity scores are for non matches. We expect the matches to have a high similarity score. So let's zoom into the right side of the distribution." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "pycharm": { "is_executing": false } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEGCAYAAACevtWaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAT/ElEQVR4nO3df7SlVX3f8fcnINBllAGZzqIzjEN0GjNtlsiahSTGhEpUwOCQVKgm1ZFOOjGLuNpKk2LtMrbLP+IyRjFpTWaBcTD+AG0so6VaMog2tmAG+S1RRoQwI8jIr4TiL/TbP86echzunXvuveeee2ff92utu+7z7Gc/5+yzuXzOnn2eZ59UFZKkvvzYYjdAkjR+hrskdchwl6QOGe6S1CHDXZI6dPhiNwDguOOOq3Xr1i12MyTpkHLDDTd8q6pWTnVsSYT7unXr2LVr12I3Q5IOKUnume6Y0zKS1CHDXZI6ZLhLUocMd0nqkOEuSR0aKdyTrEjy8SR/neSOJD+T5NgkVye5s/0+ptVNkvcm2Z3kliQnL+xLkCQdaNSR+8XAp6vqecDzgTuAi4CdVbUe2Nn2Ac4E1refrcD7xtpiSdKMZgz3JEcDPw9cClBV36uqR4BNwPZWbTtwTtveBFxWA9cBK5IcP/aWS5KmNcrI/URgH/CnSW5MckmSpwOrquq+Vud+YFXbXg3cO3T+nlb2I5JsTbIrya59+/bN/RVIkp5ilDtUDwdOBt5YVdcnuZgnp2AAqKpKMqtv/aiqbcA2gI0bN875G0POvvDKKcs/+a5Nc31ISTrkjTJy3wPsqarr2/7HGYT9N/dPt7TfD7Tje4EThs5f08okSRMyY7hX1f3AvUl+shWdDnwZ2AFsbmWbgf1D6B3A69pVM6cCjw5N30iSJmDUhcPeCHwoyRHAXcD5DN4YrkiyBbgHOK/VvQo4C9gNPN7qSpImaKRwr6qbgI1THDp9iroFXDDPdkmS5sE7VCWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nqkOEuSR0y3CWpQyOFe5K7k9ya5KYku1rZsUmuTnJn+31MK0+S9ybZneSWJCcv5AuQJD3VbEbu/6SqTqqqjW3/ImBnVa0HdrZ9gDOB9e1nK/C+cTVWkjSa+UzLbAK2t+3twDlD5ZfVwHXAiiTHz+N5JEmzNGq4F/A/k9yQZGsrW1VV97Xt+4FVbXs1cO/QuXtamSRpQg4fsd7PVdXeJH8fuDrJXw8frKpKUrN54vYmsRVg7dq1szlVkjSDkUbuVbW3/X4A+ARwCvDN/dMt7fcDrfpe4ISh09e0sgMfc1tVbayqjStXrpz7K5AkPcWM4Z7k6UmesX8beBlwG7AD2NyqbQaubNs7gNe1q2ZOBR4dmr6RJE3AKNMyq4BPJNlf/8NV9ekkfwVckWQLcA9wXqt/FXAWsBt4HDh/7K2WJB3UjOFeVXcBz5+i/EHg9CnKC7hgLK2TJM2Jd6hKUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nq0MjhnuSwJDcm+VTbPzHJ9Ul2J7k8yRGt/Mi2v7sdX7cwTZckTWc2I/d/BdwxtP8O4N1V9VzgYWBLK98CPNzK393qSZImaKRwT7IGeAVwSdsP8BLg463KduCctr2p7dOOn97qS5ImZNSR+3uA3wF+2PafBTxSVU+0/T3A6ra9GrgXoB1/tNX/EUm2JtmVZNe+ffvm2HxJ0lRmDPckvwQ8UFU3jPOJq2pbVW2sqo0rV64c50NL0rJ3+Ah1XgS8MslZwFHAM4GLgRVJDm+j8zXA3lZ/L3ACsCfJ4cDRwINjb7kkaVozjtyr6s1Vtaaq1gGvBq6pql8DPgu8qlXbDFzZtne0fdrxa6qqxtpqSdJBzec6938HvCnJbgZz6pe28kuBZ7XyNwEXza+JkqTZGmVa5v+rqmuBa9v2XcApU9T5DnDuGNomSZoj71CVpA4Z7pLUIcNdkjo0qzn3Q8nZF145Zfkn37Vpwi2RpMlz5C5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ90uPyBJkzbdsicw+aVPHLlLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdmjHckxyV5ItJbk5ye5L/2MpPTHJ9kt1JLk9yRCs/su3vbsfXLexLkCQdaJSR+3eBl1TV84GTgDOSnAq8A3h3VT0XeBjY0upvAR5u5e9u9SRJEzRjuNfAY233ae2ngJcAH2/l24Fz2vamtk87fnqSjK3FkqQZjTTnnuSwJDcBDwBXA18DHqmqJ1qVPcDqtr0auBegHX8UeNYUj7k1ya4ku/bt2ze/VyFJ+hEjhXtV/aCqTgLWAKcAz5vvE1fVtqraWFUbV65cOd+HkyQNmdXVMlX1CPBZ4GeAFUn2f9nHGmBv294LnADQjh8NPDiW1kqSRjLK1TIrk6xo238PeClwB4OQf1WrthnY/xUkO9o+7fg1VVXjbLQk6eBG+Zq944HtSQ5j8GZwRVV9KsmXgY8meTtwI3Bpq38p8MEku4GHgFcvQLslSQcxY7hX1S3AC6Yov4vB/PuB5d8Bzh1L6yRJc+IdqpLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDo1znLkmHlLMvvHLK8k++a9OEW7J4HLlLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDnkppCTN0nSXWi4ljtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ17lLWjaW01LAhrukZa/H0HdaRpI6ZLhLUodmnJZJcgJwGbAKKGBbVV2c5FjgcmAdcDdwXlU9nCTAxcBZwOPA66vqSwvTfEnL2aGwxstiGWXk/gRwYVVtAE4FLkiyAbgI2FlV64GdbR/gTGB9+9kKvG/srZYkHdSM4V5V9+0feVfV3wF3AKuBTcD2Vm07cE7b3gRcVgPXASuSHD/2lkuSpjWrOfck64AXANcDq6rqvnbofgbTNjAI/nuHTtvTyiRJEzLypZBJfhz4r8C/rqq/HUytD1RVJanZPHGSrQymbVi7du1sTpXUKefQx2ekcE/yNAbB/qGq+vNW/M0kx1fVfW3a5YFWvhc4Yej0Na3sR1TVNmAbwMaNG2f1xiBJk3Aov9nMOC3Trn65FLijqv5g6NAOYHPb3gxcOVT+ugycCjw6NH0jSZqAUUbuLwJeC9ya5KZW9u+B3wOuSLIFuAc4rx27isFlkLsZXAp5/lhbLEma0YzhXlV/CWSaw6dPUb+AC+bZLknSPLi2jKSJO5Tnsg8VLj8gSR1y5C5pwThCXzyO3CWpQ4a7JHXIcJekDhnuktShZfeB6sE+4DmUv1JLkoY5cpekDhnuktShZTctI2n8vJ596XHkLkkdcuQuSRMw3b9uFupCDsNdWqYmHTaaLKdlJKlDhrskdchwl6QOGe6S1CE/UJU0Eq9lP7Q4cpekDhnuktQhw12SOuScu7REjWuOe7Y3JTm33gfDXVpkhqkWguEudc43j+XJOXdJ6tCMI/ck7wd+CXigqv5xKzsWuBxYB9wNnFdVDycJcDFwFvA48Pqq+tLCNF1amlyQS0vBKCP3DwBnHFB2EbCzqtYDO9s+wJnA+vazFXjfeJopSZqNGcO9qj4PPHRA8SZge9veDpwzVH5ZDVwHrEhy/LgaK0kazVw/UF1VVfe17fuBVW17NXDvUL09rew+DpBkK4PRPWvXrp1jM8bLf05L6sW8P1CtqgJqDudtq6qNVbVx5cqV822GJGnIXMP9m/unW9rvB1r5XuCEoXprWpkkaYLmOi2zA9gM/F77feVQ+W8l+SjwQuDRoekbaVnzenNN0iiXQn4EOA04Lske4HcZhPoVSbYA9wDntepXMbgMcjeDSyHPX4A2S0uCYa2lbMZwr6rXTHPo9CnqFnDBfBslSZoflx+Q8Eop9cflBySpQ47cpYNwXl2HKkfuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUNeCqlZ8WYf6dBguI/AQDv0+N9My53TMpLUIUfu8+DocPHN9g5S7zjVcmG4d2ScbzaGoHRoM9y1oMb1huObjTQ7hvsysBSD0SktaWEZ7gtgoYPrUAprSYvDq2UkqUOO3JcApyie5L8ApPEw3Jcwg07SXDktI0kdMtwlqUOGuyR1yDn3CXIOXdKkOHKXpA4Z7pLUoQUJ9yRnJPlKkt1JLlqI55AkTW/s4Z7kMOA/A2cCG4DXJNkw7ueRJE1vIUbupwC7q+quqvoe8FFg+d1qKUmLaCGullkN3Du0vwd44YGVkmwFtrbdx5J8ZQHaMpPjgG8twvMeKuyfmdlHB2f/zCB/MK8+evZ0BxbtUsiq2gZsW6znB0iyq6o2LmYbljL7Z2b20cHZPzNbqD5aiGmZvcAJQ/trWpkkaUIWItz/Clif5MQkRwCvBnYswPNIkqYx9mmZqnoiyW8BnwEOA95fVbeP+3nGZFGnhQ4B9s/M7KODs39mtiB9lKpaiMeVJC0i71CVpA4Z7pLUoS7DfablD5KsTfLZJDcmuSXJWUPH3tzO+0qSl0+25ZMz1z5Ksi7Jt5Pc1H7+ePKtX3gj9M+zk+xsfXNtkjVDxzYnubP9bJ5syydnnn30g6G/oS4vuEjy/iQPJLltmuNJ8t7Wf7ckOXno2Pz/hqqqqx8GH+J+DfgJ4AjgZmDDAXW2Ab/ZtjcAdw9t3wwcCZzYHuewxX5NS6yP1gG3LfZrWAL98zFgc9t+CfDBtn0scFf7fUzbPmaxX9NS6qO2/9hiv4YJ9NHPAydP9/8LcBbwP4AApwLXj/NvqMeR+yjLHxTwzLZ9NPCNtr0J+GhVfbeqvg7sbo/Xm/n00XIwSv9sAK5p258dOv5y4OqqeqiqHgauBs6YQJsnbT59tCxU1eeBhw5SZRNwWQ1cB6xIcjxj+hvqMdynWv5g9QF13gb88yR7gKuAN87i3B7Mp48ATmzTNZ9L8uIFbeniGKV/bgZ+pW3/MvCMJM8a8dwezKePAI5KsivJdUnOWdimLlnT9eFY/oZ6DPdRvAb4QFWtYfBPow8mWa59MZ3p+ug+YG1VvQB4E/DhJM88yOP06t8Cv5DkRuAXGNyF/YPFbdKSc7A+enYNbrn/VeA9SZ6zSG3sVo+BNsryB1uAKwCq6v8ARzFY4Gi5LJ0w5z5qU1YPtvIbGMy7/sMFb/Fkzdg/VfWNqvqV9ib3llb2yCjndmI+fURV7W2/7wKuBV4wgTYvNdP14Vj+hnoM91GWP/gb4HSAJD/FILj2tXqvTnJkkhOB9cAXJ9byyZlzHyVZ2dbsJ8lPMOijuybW8smYsX+SHDf0r703A+9v258BXpbkmCTHAC9rZb2Zcx+1vjlyfx3gRcCXJ9bypWMH8Lp21cypwKNVdR/j+hta7E+UF+hT6rOArzIYVb6llf0n4JVtewPwBQZzgjcBLxs69y3tvK8AZy72a1lqfQT8U+D2VvYl4OzFfi2L1D+vAu5sdS4Bjhw6918w+DB+N3D+Yr+WpdZHwM8Ct7a/rVuBLYv9Whaofz7CYBrz+wzmzbcAbwDe0I6HwRcbfa31w8Zx/g25/IAkdajHaRlJWvYMd0nqkOEuSR0y3CWpQ4a7JHXIcNeSluSSJBtmUX9jkve27dcn+aNZPt/w+acl+dnZtVhaGsb+NXvSOFXVr8+y/i5g11yeK8nhB5x/GvAY8L/n8njjkOSwqnJZA82aI3ctCUmenuS/J7k5yW1J/lkrvzbJxrb9WJJ3Jrk9yV8kOaUdvyvJK1ud05J8aorHPzvJ9W3Bs79IsqqVvy3JB5N8gcH6Oacl+VSSdQxuOPk3bc3xFyf5epKntfOeObw/9DzntvbfnOTzreywJL/fym9J8sZWfnprz60ZrP29/67Nu5O8I8mXgHOTPCfJp5PckOR/JXneQvw3UF8cuWupOAP4RlW9AiDJ0VPUeTpwTVX9dpJPAG8HXsrgbtrtPHUJhWF/CZxaVZXk14HfAS5sxzYAP1dV305yGkBV3Z3BF5E8VlW/39p0LfAK4L8xuN3+z6vq+wc8z1uBl1fV3iQrWtlWBuvgn1SDL5A/NslRwAeA06vqq0kuA34TeE8758GqOrk9704GdzXemeSFwH9hsD66NC1H7loqbgVe2kasL66qR6eo8z3g00P1P9fC9VYG4Xkwa4DPJLkV+G3gHw0d21FV3x6hjZcA57ft84E/naLOF4APJPmXDL7QAuAXgT+pqicAquoh4CeBr1fVV1ud7Qy+3GG/ywGS/DiD2/U/luQm4E+A40doq5Y5w11LQgu5kxkE9duTvHWKat+vJ9fL+CHw3XbuD5n5X6F/CPxRVf008BsMFkLb7/+O2MYvAOva6P6wqnrK16dV1RuA/8BgVb8b8uT65bO1v00/BjxSVScN/fzUHB9Ty4jhriUhyT8AHq+qPwPeySDox+lonlw2ddTvpPw74BkHlF0GfJipR+0keU5VXV9Vb2Ww0ugJDL5J5zeSHN7qHMtgYbp1SZ7bTn0t8LkDH6+q/hb4epJz27lJ8vwR269lzHDXUvHTwBfb1MPvMphPH6e3MZjauAH41ojnfBL45f0fqLayDzH4XsuPTHPOO9sHpLcxuMrmZgbTOX8D3JLkZuBXq+o7DKZ2Ptamin4ITPdl478GbGnn3s4y+7o6zY2rQkqzkORVwKaqeu1it0U6GK+WkUaU5A+BMxmsYy4taY7cJalDzrlLUocMd0nqkOEuSR0y3CWpQ4a7JHXo/wGLDuOcFHH2SgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.hist([score for _, _, score in data if score >= 0.79], bins=50);\n", "plt.xlabel('similarity score')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Indeed, there is a cluster of scores between 0.9 and 1.0. To better visualize that these are indeed the scores for the matches, we will now extract the true_matches from the datasets and group the similarity scores into those for the matches and the non-matches (We can do this because we know the ground truth of the dataset)." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "# rec_id in dfA has the form 'rec-1070-org'. We only want the number. Additionally, as we are\n", "# interested in the position of the records, we create a new index which contains the row numbers.\n", "dfA_ = dfA.rename(lambda x: x[4:-4], axis='index').reset_index()\n", "dfB_ = dfB.rename(lambda x: x[4:-6], axis='index').reset_index()\n", "# now we can merge dfA_ and dfB_ on the record_id.\n", "a = pd.DataFrame({'ida': dfA_.index, 'rec_id': dfA_['rec_id']})\n", "b = pd.DataFrame({'idb': dfB_.index, 'rec_id': dfB_['rec_id']})\n", "dfj = a.merge(b, on='rec_id', how='inner').drop(columns=['rec_id'])\n", "# and build a set of the corresponding row numbers.\n", "true_matches = set((row[0], row[1]) for row in dfj.itertuples(index=False))" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "scores_matches = []\n", "scores_non_matches = []\n", "for (_, a), (_, b), score in data:\n", " if score < 0.79:\n", " continue\n", " if (a, b) in true_matches:\n", " scores_matches.append(score)\n", " else:\n", " scores_non_matches.append(score)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEGCAYAAACevtWaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAb8klEQVR4nO3dfXRV1b3u8e/Di1CrBVSuA4mnYIsWMRAQLEpbUY6I+AKlanuOVeTYUr20o1e9CirVtsM6cNhzqrZePVRRsFrx5fSKXq8VfEO9RQwaEERLtLQEqaQotFSsUn/3jz1Jt5iQnWQnO1l5PmNkZK255lpr7pnwZDH32nMpIjAzs2zpUuoGmJlZ8TnczcwyyOFuZpZBDnczswxyuJuZZVC3UjcA4IADDogBAwaUuhlmZh3KihUr/hQRfevb1i7CfcCAAVRWVpa6GWZmHYqk3ze0zcMyZmYZ5HA3M8sgh7uZWQYVNOYuqTdwK3AEEMC/Aa8BC4EBwHrgzIh4R5KAG4CJwLvAuRHxYtFbbmat5oMPPqCmpob33nuv1E0xoGfPnpSVldG9e/eC9yn0DdUbgEcj4nRJewF7A5cDj0fEHEmzgFnATOAkYFD6+jxwc/puZh1ETU0N++67LwMGDCB3vWalEhFs2bKFmpoaBg4cWPB+jQ7LSOoFfAm4LZ3o/YjYCkwC5qdq84HJaXkSsCBylgG9JfUr/KWYWam999577L///g72dkAS+++/f5P/F1XImPtAoBa4XdJLkm6V9EngwIjYlOr8ETgwLfcHNuTtX5PKdm/wdEmVkipra2ub1Ggza30O9vajOT+LQsK9GzACuDkihgN/JTcEUydy8wY3ae7giJgbESMjYmTfvvXeg29mZs1UyJh7DVATEc+n9fvJhftbkvpFxKY07LI5bd8IHJy3f1kqM7MO6tSLHyzq8R7690lFPV6+qqoq3nzzTSZOnLjHevvssw/bt29vtXaUWqNX7hHxR2CDpMNS0TjgFWARMDWVTQV2/fQXAecoZzSwLW/4plWdufACzlx4QVucyszaqaqqKh555JFSN6PkCr3P/TvAXZJWARXANcAc4ARJ64B/TusAjwBvANXAz4H/XtQWm1nmrV+/ns997nOce+65HHrooZx11lksWbKEMWPGMGjQIJYvX87y5cs5+uijGT58OMcccwyvvfYa77//PldeeSULFy6koqKChQsXsn37dqZNm0Z5eTlDhw7lgQceqDvPFVdcwbBhwxg9ejRvvfUWALW1tXzlK19h1KhRjBo1iueeew6Ap59+moqKCioqKhg+fDh/+ctfStI3hSroVsiIqAJG1rNpXD11A5jRwnaZWSdXXV3Nfffdx7x58xg1ahR33303zz77LIsWLeKaa65hwYIFPPPMM3Tr1o0lS5Zw+eWX88ADD/DDH/6QyspKfvaznwEwc+ZMevXqxcsvvwzAO++8A8Bf//pXRo8ezY9+9CMuvfRSfv7znzN79my++93vcuGFF/KFL3yBP/zhD5x44omsXbuWH//4x9x0002MGTOG7du307Nnz5L1TSHaxcRhZma7GzhwIOXl5QAMGTKEcePGIYny8nLWr1/Ptm3bmDp1KuvWrUMSH3zwQb3HWbJkCffcc0/dep8+fQDYa6+9OOWUUwA48sgjWbx4cV39V155pa7+n//8Z7Zv386YMWO46KKLOOuss5gyZQplZWWt8rqLxdMPmFm71KNHj7rlLl261K136dKFnTt38r3vfY/jjjuO1atX89BDDzX5PvDu3bvX3WLYtWtXdu7cCcCHH37IsmXLqKqqoqqqio0bN7LPPvswa9Ysbr31Vnbs2MGYMWN49dVXi/RKW4fD3cw6pG3bttG/f+4jNHfccUdd+b777vuR8fATTjiBm266qW5917BMQ8aPH89Pf/rTuvWqqioAXn/9dcrLy5k5cyajRo1q9+HuYRkza1Rr3rrYXJdeeilTp07l6quv5uSTT64rP+6445gzZw4VFRVcdtllzJ49mxkzZnDEEUfQtWtXrrrqKqZMmdLgcW+88UZmzJjB0KFD2blzJ1/60pe45ZZbuP7663nyySfp0qULQ4YM4aSTTmqLl9lsyr3/WVojR46MYjysY9dtkPd+9eYWH8usM1u7di2DBw8udTMsT30/E0krIqK+m108LGNmlkUOdzOzDHK4m5llkMPdzCyDHO5mZhnkcDczyyDf525mjSr2bKsd9Xbla665hssvv3yPdc4991xOOeUUTj/99DZqVf185W5mVqBrrrmm1E0omMPdzNqd9evXM3jwYL75zW8yZMgQxo8fz44dO6iqqmL06NEMHTqUL3/5y3VTCYwdO5aZM2dy1FFHceihh/LMM8/Ue9yxY8dy4YUXMnLkSAYPHswLL7zAlClTGDRoELNnz66rN3nyZI488kiGDBnC3LlzAZg1axY7duygoqKCs846C4AFCxYwdOhQhg0bxtlnn123/9KlSznmmGM45JBDuP/+++vKr7vuOkaNGsXQoUO56qqrgNzslCeffDLDhg3jiCOOYOHChUXpQ4e7mbVL69atY8aMGaxZs4bevXvzwAMPcM4553DttdeyatUqysvL+cEPflBXf+fOnSxfvpzrr7/+I+W722uvvaisrOT8889n0qRJ3HTTTaxevZo77riDLVu2ADBv3jxWrFhBZWUlN954I1u2bGHOnDl84hOfoKqqirvuuos1a9Zw9dVX88QTT7By5UpuuOGGunNs2rSJZ599locffphZs3JPJX3sscdYt24dy5cvp6qqihUrVrB06VIeffRRDjroIFauXMnq1auZMGFCUfrP4W5m7dLAgQOpqKgAclPyvv7662zdupVjjz0WgKlTp7J06dK6+rvmiznyyCNZv359g8c97bTTACgvL2fIkCH069ePHj16cMghh7BhwwYgN7/Mrod4bNiwgXXr1n3sOE888QRnnHEGBxxwAAD77bdf3bbJkyfTpUsXDj/88LqHgDz22GM89thjDB8+nBEjRvDqq6+ybt06ysvLWbx4MTNnzuSZZ56hV69eze2yj/AbqmbWLuVP+du1a1e2bt1aUP386XunTZvGSy+9xEEHHVT36L38qYN3n1Z4586dPPXUUyxZsoTf/OY37L333owdO7bJ0wnnH3fX/F0RwWWXXca3vvWtj9V/8cUXeeSRR5g9ezbjxo3jyiuvbNL56uMrdzPrEHr16kWfPn3qxtPvvPPOuqv4htx+++1Nfqbqtm3b6NOnD3vvvTevvvoqy5Ytq9vWvXv3uoeCHH/88dx33311Qzlvv/32Ho974oknMm/evLqHcm/cuJHNmzfz5ptvsvfee/P1r3+dSy65hBdffLHgtu6Jr9zNrFHt5dbF+fPnc/755/Puu+9yyCGHcPvttxf9HBMmTOCWW25h8ODBHHbYYYwePbpu2/Tp0xk6dCgjRozgrrvu4oorruDYY4+la9euDB8+/CPzyu9u/PjxrF27lqOPPhqAffbZh1/84hdUV1dzySWX0KVLF7p3787NNxenrz3lr5l9jKf8bX885a+ZmTnczcyyyOFuZvVqD0O2ltOcn4XD3cw+pmfPnmzZssUB3w5EBFu2bKFnz55N2s93y5jZx5SVlVFTU0NtbW2pm2Lk/tiWlZU1aR+Hu5l9TPfu3Rk4cGCpm2EtUNCwjKT1kl6WVCWpMpXtJ2mxpHXpe59ULkk3SqqWtErSiNZ8AWZm9nFNGXM/LiIq8u6pnAU8HhGDgMfTOsBJwKD0NR3wTedmZm2sJW+oTgLmp+X5wOS88gWRswzoLalfC85jZmZNVGi4B/CYpBWSpqeyAyNiU1r+I3BgWu4PbMjbtyaVfYSk6ZIqJVX6TRszs+Iq9A3VL0TERkn/DVgs6dX8jRERkpp0z1REzAXmQm76gabsa2Zme1bQlXtEbEzfNwO/Ao4C3to13JK+b07VNwIH5+1elsrMzKyNNBrukj4pad9dy8B4YDWwCJiaqk0FHkzLi4Bz0l0zo4FtecM3ZmbWBgoZljkQ+JWkXfXvjohHJb0A3CvpPOD3wJmp/iPARKAaeBeYVvRWm5nZHjUa7hHxBjCsnvItwLh6ygOYUZTWmZlZs3huGTOzDHK4m5llkMPdzCyDHO5mZhnkcDczyyCHu5lZBjnczcwyyOFuZpZBDnczswxyuJuZZZDD3cwsgxzuZmYZ5HA3M8sgh7uZWQY53M3MMsjhbmaWQQ53M7MMcribmWWQw93MLIMc7mZmGeRwNzPLIIe7mVkGOdzNzDLI4W5mlkEOdzOzDHK4m5llUMHhLqmrpJckPZzWB0p6XlK1pIWS9krlPdJ6ddo+oHWabmZmDWnKlft3gbV569cCP4mIzwLvAOel8vOAd1L5T1I9MzNrQwWFu6Qy4GTg1rQu4Hjg/lRlPjA5LU9K66Tt41J9MzNrI4VeuV8PXAp8mNb3B7ZGxM60XgP0T8v9gQ0Aafu2VP8jJE2XVCmpsra2tpnNNzOz+jQa7pJOATZHxIpinjgi5kbEyIgY2bdv32Ie2sys0+tWQJ0xwGmSJgI9gU8BNwC9JXVLV+dlwMZUfyNwMFAjqRvQC9hS9JabmVmDGr1yj4jLIqIsIgYAXwOeiIizgCeB01O1qcCDaXlRWidtfyIioqitNjOzPWrJfe4zgYskVZMbU78tld8G7J/KLwJmtayJZmbWVIUMy9SJiKeAp9LyG8BR9dR5DzijCG0zM7Nm8idUzcwyyOFuZpZBDnczswxyuJuZZZDD3cwsgxzuZmYZ5HA3M8ugThHuZy68gDMXXlDqZpiZtZlOEe5mZp2Nw93MLIMc7mZmGeRwNzPLIIe7mVkGOdzNzDLI4W5mVmSnXvwgp178YOMVW5HD3cwsgxzuZmYZ5HA3M8sgh7uZWQY53M3MMsjhbmaWQQ53M7MMcribmWWQw93MLIMc7mZmGeRwNzPLoEbDXVJPScslrZS0RtIPUvlASc9Lqpa0UNJeqbxHWq9O2we07kswM7PdFXLl/jfg+IgYBlQAEySNBq4FfhIRnwXeAc5L9c8D3knlP0n1zMysDTUa7pGzPa12T18BHA/cn8rnA5PT8qS0Tto+TpKK1mIzM2tUQWPukrpKqgI2A4uB14GtEbEzVakB+qfl/sAGgLR9G7B/PcecLqlSUmVtbW3LXoWZmX1EQeEeEX+PiAqgDDgK+FxLTxwRcyNiZESM7Nu3b0sPZ2ZmeZp0t0xEbAWeBI4GekvqljaVARvT8kbgYIC0vRewpSitNTOzghRyt0xfSb3T8ieAE4C15EL+9FRtKrDrsSOL0jpp+xMREcVstJmZ7Vm3xqvQD5gvqSu5Pwb3RsTDkl4B7pF0NfAScFuqfxtwp6Rq4G3ga63QbjMz24NGwz0iVgHD6yl/g9z4++7l7wFnFKV1ZmbWLP6EqplZBjnczcwyyOFuZpZBDnczswxyuJuZZZDD3cwsgxzuZmYZVMiHmMzMOpxTL36wbvmhf59UwpaUhq/czcwyyOFuZpZBDnczswxyuJuZZZDD3cwsgxzuZmbNdOrFD37krpz2xOFuZpZBDnczswxyuJuZZZDD3cwsgxzuZmYZ5HA3M8sgh7uZWQY53M3MMshT/ppZp9JZpgJ2uJuZkb3Q97CMmVkGOdzNrENrz/O7lFKj4S7pYElPSnpF0hpJ303l+0laLGld+t4nlUvSjZKqJa2SNKK1X4SZmX1UIVfuO4GLI+JwYDQwQ9LhwCzg8YgYBDye1gFOAgalr+nAzUVvtZmZ7VGj4R4RmyLixbT8F2At0B+YBMxP1eYDk9PyJGBB5CwDekvqV/SWm5lZg5o05i5pADAceB44MCI2pU1/BA5My/2BDXm71aSy3Y81XVKlpMra2tomNtvMsqqhMXSPrTdNweEuaR/gAeB/RMSf87dFRADRlBNHxNyIGBkRI/v27duUXc3MrBEF3ecuqTu5YL8rIv4rFb8lqV9EbErDLptT+Ubg4Lzdy1KZmVmH01Hvfy/kbhkBtwFrI+I/8jYtAqam5anAg3nl56S7ZkYD2/KGb8zMrA0UcuU+BjgbeFlSVSq7HJgD3CvpPOD3wJlp2yPARKAaeBeYVtQWm5lZoxoN94h4FlADm8fVUz+AGS1sl5mZtYA/oWpmJeG7X1qXw93MLIMc7mbWqnyFXhoOdzOzDHK4m5llkMPdzCyDHO5mZhnkcDczyyCHu5lZBnXqcD9z4QWcufCCUjfDzKzoCpoV0sxsTzrqzIlZ1qmv3M3M2lJbfqDL4W5mlkEeljHrxDyckl2+cjczyyCHu5lZBjnczcwyyOFuZpZBfkPVzArmN2A7Dl+5m5llkMPdzCyDHO5m7VhDn2hsarl1Ph5zN7OP8dh6x+crd7N2wFfcVmwOd7NOwH88Oh8Py5gVmYc0rD1o9Mpd0jxJmyWtzivbT9JiSevS9z6pXJJulFQtaZWkEa3ZeDMzq18hwzJ3ABN2K5sFPB4Rg4DH0zrAScCg9DUduLk4zTQzs6ZoNNwjYinw9m7Fk4D5aXk+MDmvfEHkLAN6S+pXrMaamVlhmvuG6oERsSkt/xE4MC33Bzbk1atJZR8jabqkSkmVtbW1zWyGmZnVp8V3y0REANGM/eZGxMiIGNm3b9+WNsPMzPI0N9zf2jXckr5vTuUbgYPz6pWlMjMza0PNvRVyETAVmJO+P5hX/m1J9wCfB7blDd90GGcuvKBu+d6v+j1hKw7fImltqdFwl/RLYCxwgKQa4CpyoX6vpPOA3wNnpuqPABOBauBdYFortNmsXXBYW3vWaLhHxL80sGlcPXUDmNHSRpmZWct4+gGzxB/RtyxxuJuZZZDnljFrhMfWrSPylbuZWQY53M3MMsjhbmaWQQ53M7MMcribmWWQw93MLIMc7tYk/qCPWcfgcLfM8h8i68wc7mZmGeRwtw6vqVfovqK3zsDhnjHFCi4HoFnH5rllmqAjP8RjV1C39dwoxZyXpVSvwawjcrhbSXgyLrPW5XDv5Ip1NeywNmtfHO4dkIPUzBrjcLd2xX+4zIrD4d4OONDMrNgc7kWw+100Hsc2s1JzuLfArvD9xFElboiZ2W78ISYzswxyuJuZZZCHZQrg4Rcz62gyG+75b0aWKpQ78nQFZtaxtUq4S5oA3AB0BW6NiDmtcZ7mauhKvLWv0D/6B+dRwKFvZq2j6OEuqStwE3ACUAO8IGlRRLxS7HNB+7hCL6ZdV/sOfTNrida4cj8KqI6INwAk3QNMAlol3DuLhkK/qeWNHX/3fTy0ZNYxKSKKe0DpdGBCRHwjrZ8NfD4ivr1bvenA9LR6GPBaURvSuAOAP7XxOTsa91Hj3EeNcx/tWUv659MR0be+DSV7QzUi5gJzS3V+SZURMbJU5+8I3EeNcx81zn20Z63VP61xn/tG4OC89bJUZmZmbaQ1wv0FYJCkgZL2Ar4GLGqF85iZWQOKPiwTETslfRv4NblbIedFxJpin6cISjYk1IG4jxrnPmqc+2jPWqV/iv6GqpmZlZ7nljEzyyCHu5lZBmUy3CVNkPSapGpJs+rZ/k+SnpT0kqRVkibmbbss7feapBPbtuVtp7l9JGmApB2SqtLXLW3f+tZXQP98WtLjqW+eklSWt22qpHXpa2rbtrzttLCP/p73O5TJGy4kzZO0WdLqBrZL0o2p/1ZJGpG3reW/QxGRqS9yb+K+DhwC7AWsBA7frc5c4IK0fDiwPm95JdADGJiO07XUr6md9dEAYHWpX0M76J/7gKlp+XjgzrS8H/BG+t4nLfcp9WtqT32U1reX+jW0QR99CRjR0L8XYCLwfwEBo4Hni/k7lMUr97rpDyLifWDX9Af5AvhUWu4FvJmWJwH3RMTfIuJ3QHU6Xta0pI86g0L653DgibT8ZN72E4HFEfF2RLwDLAYmtEGb21pL+qhTiIilwNt7qDIJWBA5y4DekvpRpN+hLIZ7f2BD3npNKsv3feDrkmqAR4DvNGHfLGhJHwEMTMM1T0v6Yqu2tDQK6Z+VwJS0/GVgX0n7F7hvFrSkjwB6SqqUtEzS5NZtarvVUB8W5Xcoi+FeiH8B7oiIMnL/NbpTUmfti4Y01EebgH+KiOHARcDdkj61h+Nk1f8EjpX0EnAsuU9h/720TWp39tRHn47cR+7/Fbhe0mdK1MbMymKgFTL9wXnAvQAR8RugJ7nJezrL1AnN7qM0ZLUlla8gN+56aKu3uG012j8R8WZETEl/5K5IZVsL2TcjWtJHRMTG9P0N4ClgeBu0ub1pqA+L8juUxXAvZPqDPwDjACQNJhdctane1yT1kDQQGAQsb7OWt51m95GkvmnOfiQdQq6P3mizlreNRvtH0gF5/9u7DJiXln8NjJfUR1IfYHwqy5pm91Hqmx676gBj6JxTgi8Czkl3zYwGtkXEJor1O1Tqd5Rb6V3qicBvyV1VXpHKfgiclpYPB54jNyZYBYzP2/eKtN9rwEmlfi3trY+ArwBrUtmLwKmlfi0l6p/TgXWpzq1Aj7x9/43cm/HVwLRSv5b21kfAMcDL6XfrZeC8Ur+WVuqfX5IbxvyA3Lj5ecD5wPlpu8g92Oj11A8ji/k75OkHzMwyKIvDMmZmnZ7D3cwsgxzuZmYZ5HA3M8sgh7uZWQY53K1dk3SrpMObUH+kpBvT8rmSftbE8+XvP1bSMU1rsVn7UPTH7JkVU0R8o4n1K4HK5pxLUrfd9h8LbAf+X3OOVwySukaEpzWwJvOVu7ULkj4p6f9IWilptaSvpvKnJI1My9slXSdpjaQlko5K29+QdFqqM1bSw/Uc/1RJz6cJz5ZIOjCVf1/SnZKeIzd/zlhJD0saQO4DJxemOce/KOl3krqn/T6Vv553njNS+1dKWprKukr6cSpfJek7qXxcas/Lys39vetTm+slXSvpReAMSZ+R9KikFZKekfS51vgZWLb4yt3aiwnAmxFxMoCkXvXU+STwRERcIulXwNXACeQ+TTufj0+hkO9ZYHREhKRvAJcCF6dthwNfiIgdksYCRMR65R5Esj0ifpza9BRwMvC/yX3c/r8i4oPdznMlcGJEbJTUO5VNJzcPfkXkHiC/n6SewB3AuIj4raQFwAXA9WmfLRExIp33cXKfalwn6fPA/yI3P7pZg3zlbu3Fy8AJ6Yr1ixGxrZ467wOP5tV/OoXry+TCc0/KgF9Lehm4BBiSt21RROwooI23AtPS8jTg9nrqPAfcIemb5B5oAfDPwH9GxE6AiHgbOAz4XUT8NtWZT+7hDrssBJC0D7mP698nqQr4T6BfAW21Ts7hbu1CCrkR5IL6aklX1lPtg/jHfBkfAn9L+35I4/8L/Snws4goB75FbiK0Xf5aYBufAwakq/uuEfGxx6dFxPnAbHKz+q3QP+Yvb6pdbeoCbI2Iiryvwc08pnUiDndrFyQdBLwbEb8AriMX9MXUi39Mm1roMyn/Auy7W9kC4G7qv2pH0mci4vmIuJLcTKMHk3uSzrckdUt19iM3Md0ASZ9Nu54NPL378SLiz8DvJJ2R9pWkYQW23zoxh7u1F+XA8jT0cBW58fRi+j65oY0VwJ8K3Och4Mu73lBNZXeRe67lLxvY57r0BulqcnfZrCQ3nPMHYJWklcC/RsR75IZ27ktDRR8CDT1s/CzgvLTvGjrZ4+qseTwrpFkTSDodmBQRZ5e6LWZ74rtlzAok6afASeTmMTdr13zlbmaWQR5zNzPLIIe7mVkGOdzNzDLI4W5mlkEOdzOzDPr/wUIK/pP1YT4AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.hist([scores_matches, scores_non_matches], bins=50, label=['matches', 'non-matches'])\n", "plt.legend(loc='upper right')\n", "plt.xlabel('similarity score')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "We can see that the similarity scores for the matches and the ones for the non-matches form two different distributions. With a suitable linkage schema, these two distributions hardly overlap. \n", "\n", "When choosing a similarity threshold for solving, the valley between these two distributions is a good starting point. In this example, it is around 0.82. We can see that almost all similarity scores above 0.82 are from matches, thus the solver will produce a linkage result with high precision. However, recall will not be optimal, as there are still some scores from matches below 0.82. By moving the threshold to either side, you can favour either precision or recall." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[31mProject deleted\u001b[0m\r\n" ] } ], "source": [ "# Deleting the project\n", "!anonlink delete-project --project=\"{credentials['project_id']}\" \\\n", " --apikey=\"{credentials['result_token']}\" \\\n", " --server=\"{url}\"" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.0" } }, "nbformat": 4, "nbformat_minor": 4 }